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A growing sense of urgency regarding the need to reform 
present patterns of delivering human services to children 
and families underscores the need for systems to have 
mechanisms that ensure accountability. One strong element 
of this reform is that human services need to become 
accountable for achieving measurable outcomes rather than 
continuing to focus on technical compliance with rules or 
on simple demonstration of service need. Doug Nelson, 
Executive Director of the Annie E. Casey Foundation notes, 
"It has become a well-worn observation that success in 
human services is too often measured by persons served or 
services provided and too rarely by results achieved. 
Difficult though it may be, the reform required is clear. 
Helping agencies, service programs and schools need to be 
held genuinely accountable for progress on specific, 
publicly articulated and accurately tracked outcomes for the 
children and families they serve." (Nelson, 1993). The 
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summaries that follow describe an evolutionary process for 
incorporating use of outcome information into planning and 
program development. 

The opening summary of this symposium presents the 
Ecology of Outcomes framework, an overall framework 
designed to guide the development of outcome information 
systems. This framework provides a foundation for utilizing 
outcome information in ways that provide opportunities for 
learning and self-correction; it emphasizes using outcome 
information to inform decisions that shape service planning 
and delivery. As discussed below, components of the 
Ecology of Outcomes framework include the Principles for 
Outcome Accountability, Prerequisites and Building Blocks 
for building outcome accountability, guidelines for 
Implementing an Outcome Information System, and an 
approach for Utilizing the Results. This framework was 
developed through the System Accountability Study, an 
initiative of the federally funded Research and Training 
Center for Children's Mental Health. 

During the symposium, three presentations described 
initiatives that exemplify the principles and components of 
the Ecology of Outcomes. These included development of 
the statewide information system for the Texas Children's 
Mental Health Plan; lessons learned from the 
Pennsylvania's SumOne for Kids, developed by Pressley 
Ridge School; and California's system of care approach to 
outcome tracking. Summaries of the Texas and Pensylvania 
activities are presented in this volume. 

The Ecology of Outcomes: System Accountability in 
Children's Mental Health 

Mario Hernandez, Ph.D. & Sharon Hodges, M.B.A. 

Introduction 

A fundamental reason for tracking outcomes in applied 
service settings is to determine whether the person receiving 
services benefits in an observable manner as a result of the 
services provided. For the purposes of this summary, 
outcomes are defined as the results or the impact of services 
provided to children and their families. Furthermore, 
outcome accountability can be defined as the systems of 
care's responsibility for accomplishing publicly articulated 
goals of service provision, as measured through accurate 
monitoring over time. 

The purpose of this summary is to present a framework that 
can serve as the foundation for utilization of outcome 
information to provide opportunities for learning and 
self-correction. The four sets of components of the Ecology 
of Outcomes framework include 1) Principles for Outcomes 
Accountability; 2) Prerequisites and Building Blocks for 
building outcome accountability; 3) guidelines for 
Implementing an Outcome Information System; and 4) an 
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approach for Utilizing the Results. The Ecology 
framework's emphasis on outcome information as a 
resource for use in decision making is evident in the 
Principles for Outcome Accountability. The next sections of 
this summary present the principles for outcome 
accountability, and then discusses each component of the 
framework. 

Principles for Outcome Accountability 

Stroul and Friedman (1986) recognized that although the 
components and organizational structure of a child-serving 
system might vary from state to state or community to 
community, the development and implementation of a 
system of care should be guided by a set of values and 
principles. Similarly, there are principles central to the 
successful development and integration of outcome 
information into the planning and delivery of services that 
transcend the variability and unique characteristics of any 
child-serving system's components and organizational 
structure. Ten guiding principles have been identified for 
the development and utilization of outcome information in 
systems of care (Hernandez & Hodges, 1996). These 
principles, listed below, are central to the design and 
operation of an outcome accountability approach because 
they specify what drives and shapes the development and 
implementation of the framework. 

1. Outcome information cannot be collected in isolation of 
information about who is served and what services are 
offered. 

2. Outcome information should be used to improve service 
planning and delivery 

3. Outcome information should be relevant and accessible to 
key stakeholders in the system of care. 

4.The application and use of outcome information should be 
informed by the available research base. 

5. Outcome information should support culturally competent 
decision making in service planning and delivery. 

6. Key stakeholders should be involved in identifying and 
selecting outcomes to be measured. 

7. Data elements for the outcome measurements should be 
clearly defined. 

8. Outcome information should be useful to managers, 
administrators, and direct service providers. 

9. The process for collecting, analyzing and communicating 
outcome results should be timely and occur on a predictable 
schedule. 
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10. Outcome information should provide the opportunity 
for corrective action. 

Prerequisites and Building Blocks 

Prerequisites and Building Blocks, discussed below, are two 
essential components of successful implementation of an 
outcome accountability approach. They should be seen 
together as laying a foundation on which accountability can 
be built and thrive (see Figure 1) . 

The purpose of the Prerequisite component is to assess a 
service system's level of commitment to building an 
outcome accountability system. In essence, the Prerequisite 
phase should be used to determine if there is enough 
momentum and motivation to establish and sustain an 
outcome system. Two aspects, leadership and political 
climate are key to the Prerequisite phase of the Ecology of 
Outcomes framework. 

The Building Blocks component represents the 
development of a plan for building and implementing an 
outcome system. The primary tasks for this component 
involve clarifying the reasons for developing an outcome 
system, describing what needs to be accomplished in order 
to implement accountability, and determining baseline 
levels to establish the current status of relevant data. The 
primary aspects of the Building Blocks phase of 
development are (a) establishing a process for involving 
stakeholders; (b) clarifying the language of outcome and 
accountability; (c) assessing current capacity for building a 
system of outcome accountability; and (d) planning for 
implementation. 

The significant challenge inherent in both components is 
building a shared vision among stakeholders about what 
shape the accountability approach will take when complete. 
While this can be a formidable task, if not addressed, it can 
lead to a breakdown in the development process 
(Meadowcroft, Pierce, and Beck, 1994). 

When successfully completed, activities described by the 
aspects of Building Blocks and Prerequisites components 
yield consensus among key stakeholders about who the 
system hopes to serve, what services are expected to be 
provided, and what outcomes the system hopes to produce. 
All three elements of consensus are critical to a fully 
functioning and useful accountability approach (Usher, 
1993a,b). If only outcomes were tracked and reported, it 
would be impossible to use the information to improve 
service delivery; that is information and data without 
context, purpose, and interpretation is useless. 

The Outcome Information Components 

The Ecology framework maintains that outcome 
information cannot be used in isolation of information 
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about who is being served and what services are being 
provided. From this perspective there are three components 
to the outcome information framework: 1) populations 
targeted for services; and 2) what services are provided; and 
3) information about what outcomes have been achieved. 

With respect to answering questions of who is being served, 
the Ecology framework suggests that tests that two broad 
categories of information about children and families will 
be useful in service planning and delivery: 1) information 
about children and families that makes it possible to 
determine whether the system serves the children and 
families it intended to serve; and 2) other information about 
child and family characteristics that may influence the 
system's outcomes. 

Fundamentally, service providers and other stakeholders 
need to know that the populations they intend to serve are, 
in fact, being served. A system which fails to serve its 
intended population cannot accurately assess its outcomes. 
In addition to information about target population 
characteristics, information about other child and family 
characteristics can be useful in the interpretation of what 
may have influenced an achieved outcome. Bums (in press) 
provides a list of suggested child and family characteristics 
which may influence outcomes. These include risk factors 
such as poverty, family history of mental illness; illness 
severity, chronicity and co-morbidity; family strengths and 
tolerance of stress; social support; family member's case 
management skills; and treatment adherence by family 
members and therapists. Combining a limited number of 
carefully chosen child and family characteristics with 
information about whether target populations have been 
served can greatly enhance a system's ability to interpret its 
outcomes more confidently. 

In considering how to describe services that are being 
provided, the Ecology framework suggests that four aspects 
are useful for tracking and monitoring services, as well as 
interpreting future outcome information (Hernandez & 
Goldman, 1996). These are: 1) intensity, frequency and 
duration of services; 2) location of services; 3) variety and 
sequencing of services; and, 4) integrity of services. 

This approach requires that planners in organizations 
articulate, in operational terms, what services they expect to 
offer. This operationalization and tracking of service 
aspects gives service systems personnel confidence that the 
results of their efforts can be plausibly related to the type of 
services they provide (Dym, 1 996). The Ecology 
framework sees outcome information as a measure of what 
the system has accomplished. It should be emphasized that 
the Ecology framework stresses the use of outcomes in the 
context of managerial needs rather than for purposes of 
generalizability and application to larger social contexts. It 
is necessary to establish specific criteria for the selection of 
outcomes to be measured. The Ecology framework uses a 
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series of questions, shown below, to guide the selection of 
outcomes. 

• Is the outcome information useful to managers and 
administrators? 

• Is the outcome information useful to front-line 
workers? 

• Do the outcome results provide opportunity for 
corrective action? 

• Do the outcome results support the achievement of 
cultural competence? 

Once outcome domains have been selected, a second layer 
of decision making has to occur in order to select the 
indicators. That is, decisions must be made as to what 
indicators will be used to measure the outcomes and what 
criteria should be applied in making the selection. The 
Ecology framework offers several questions which may be 
useful in selecting the indicators. These are shown below: 

• Does the indicator adequately represent the status of 
an outcome? 

• Is the indicator easily measured? 

• Is the process of data collection and reporting realistic 
and sustainable? 

• Does the indicator provide valid and reliable 
information about the outcome? 

Utilizing the Results 

The Ecology framework assumes that using outcome 
information is a process, not an event (Bums, in press). 

Two primary elements in the process of utilizing the results 
rest on this assumption. These elements are 1) the process 
of interpreting the outcome information, and 2) action 
decisions made as a result of what has been learned. Figure 
2 illustrates that output is produced by the child-serving 
system in the form of system information regarding who the 
system has served, what services have been provided, and 
what outcomes have been produced. 

This interpretive process requires returning to the work 
generated in the Building Blocks component and measuring 
outcomes against goals that were developed for the service 
system. The interrelationships among who was served, what 
services were provided, and what outcomes have resulted 
must be considered in the interpretive process. Baseline 
information about all elements at the beginning of the 
measurement period becomes the reference for 
understanding the meaning of the information and results. 

The focus of the action step is on modifying service 
planning and delivery, as needed, based on an assessment of 
the status of the results. This use of the interpretive process 
to inform a decision to either change or not change aspects 
of service planning and delivery is best understood as a 
process of working toward improved results rather than an 
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end result. Rather than a static, one-time process, a system 
of utilizing the results should be embedded into day-to-day 
management. 

Implications for Children's Mental Health 

The shift in interest toward results-based accountability 
raises hopes that mental health systems will respond more 
flexibly to those they serve, that public faith in the ability of 
human service institutions to accomplish their intended 
purposes will be restored, and that communities will be 
better able to plan their support of children and families 
(Schorr et al. 1994). A System of Care for Severely 
Disturbed Children and Youth (Stroul & Friedman, 1986) 
more clearly defined the concept of system of care and 
provided guidance in how to build systems that would 
allow children to receive services while remaining at home 
and in their communities. We believe the Ecology of 
Outcomes framework will both complement and expand the 
systems of care concept by helping policy makers and 
administrators establish strategies to build outcome 
information systems and incorporate outcome information 
into decisions that impact the planning and delivery of 
services to children and their families. 
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The Development of a Continuous Evaluation System 
for the Texas Children's Mental Health Plan: Building 
an Evaluation-Stakeholder Feedback Loop 

Lawrence W. Rouse, Ph.D. 

Introduction 

This summary focuses on the development of an ongoing 
evaluation system associated with the Texas Children's 
Mental Health Plan (TCMHP) and the steps taken to build a 
feedback loop between evaluators and stakeholders. The 
Texas Children's Mental Health Plan is an interagency 
effort to develop a continuum of community-based mental 
health services for children, adolescents, and their families 
based on the federal Children and Adolescent Services 
System Program Model (CASSP). The ongoing evaluation 
was developed in order to provide children's mental health 
administrators at the state and local level with information 
about children's mental health services and demonstrate 
accountability to the consumers and funding sources. 

The purpose of this summary is to: (a) explain how 
outcomes are important to the basic philosophy of TCMHP; 
(b) describe the system for providing information to 
st^eholders (e.g., consumers, service providers, program 
managers, advocates, and legislators); and (c) describe the 
interactions that have taken place between evaluators and 
various TCMHP stakeholders in creating an ongoing stream 
of evaluation information for decision making. 
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The Texas Children's Mental Health Plan 

In 1992, the Texas legislature appropriated monies to the 
Texas Department of Mental Health and Mental Retardation 
(TXMHMR) for the implementation of the Texas Children's 
Mental Health Plan. The primary goal of the TCMHP is to 
develop and implement a public community-based mental 
health system for children, adolescents, and their families 
through the coordination of resources of all the state 
child-serving agencies. An essential feature of the TCMHP 
is the participation, at the state and local level, of the 
child-serving agencies, advocates, and consumers in 
management teams with the express purpose of making 
collaborative decisions about TCMHP activities. TCMHP 
services are organized into three components: (a) "core" 
mental health services, (b) services to children referred from 
the juvenile justice system, and (c) early intervention and 
prevention services. During FY '95, a total of 26,000 
children were served through TCMHP services. 

Another essential feature of the plan is a list of outcomes to 
be measured for each of the components of the TCMHP. 

The outcomes were written into the plan from the very 
beginning to assure the stakeholders, as well as the state 
legislature, that the effectiveness of the services were being 
« measured and that decisions about the TCMHP were being 
assisted by evaluation data. In addition, TXMHMR has 
been committed to the implementation of the principles of 
Total Quality Management (TQM) as a work philosophy. 
One of the hallmarks of the TQM approach is the 
measurement of work activities, using this information in 
modifying work processes to increase productivity and 
effectiveness. Finally, in its shift towards a managed care 
organizational mode, TXMHMR has recently begun 
incorporating "outcomes to be attained" in its contracts with 
each of the community service sites. Therefore, the 
collection and dissemination of evaluation information to 
TXMHMR and community site managers is central to the 
continual development of the TCMHP. 

Description of the Evaluation 

The development of a continuous evaluation system for the 
TCMHP was characterized by three stages that began in 
1990. The first stage was a summative evaluation of the 
impact of one main service type on child and family 
functioning at five mental health centers. In 1992, the 
evaluation was extended to 1 6 sites and included the 
ongoing evaluation of 14 service types. By 1994, the 
evaluation was implemented at all 45 community mental 
health authorities across the state. The evaluation currently 
involves all children served in the public mental health 
system and 1 8 service types. 

The evaluation is managed by a committee of professional 
evaluation personnel representing the nine state agencies 
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which participate in the TCMHP, representatives from 
consumer advocate groups, program directors from the 
sites, and a parent representing the viewpoint of the 
consumers. 

The basic evaluation design is characterized by 
pretreatment, post-treatment, and follow-up measurements 
of consumer demographics, history, outcomes, and 
satisfaction with services. Outcomes of services include 
general psychological functioning, behavioral-emotional 
functioning, out-of-home placement rates, and 
social-community functioning. A multi-method/multi-rater 
approach to measurement is employed to collect 
information from the service providers, children, parents, 
and collateral providers using interviews, checklists, and 
rating scales. Data is collected primarily by the service 
providers at pre-and post-treatment. Follow-up data is 
collected by the TXMHMR research office. 

Communicating evaluation results has been accomplished 
through the establishment of the Quarterly Service Report, 
Quarterly Report Review, and special reports. The 
Quarterly Service Report provides a summary of key 
indicators in the areas of numbers served, demographics, 
and outcome measures. The Quarterly Report Review is 
used as a vehicle for a discussion of the figures on the 
Quarterly Service Report and also provides an opportunity 
for the publishing of other evaluation results that may be 
relevant. Special reports are also produced for TXMHMR 
center managers and other stakeholders as requested. A 
catalogue of special reports is maintained and available for 
reference. Taken together, these publications are seen as an 
essential tool in facilitating a most important set of 
evaluation-stakeholder transactions (i.e., a feedback loop). 

Evaluator-Stakeholder Feedback Loop 

The feedback loop for the TCMHP is characterized by at 
least four major activities that are constantly being 
reiterated. The loop begins as stakeholders raise questions 
about the program. Evaluators collect and analyze data in 
response to these questions. Evaluators then collaborate 
with stakeholders in using results, and stakeholders use 
conclusions about results in making decisions, which lead 
to new questions, beginning the cycle again. 

Within each of these activities, interactions between 
evaluators and stakeholders provide inertia for the 
reiteration of the feedback loop. Throughout the 
development of the TCMHP evaluation, these interactions 
have contributed to modifications in major activities in 
order to meet the needs of both the evaluators and the 
stakeholders. 

Stakeholders Raise Initial Questions 

The provision of guidelines for evaluation of TCMHP 
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outcomes created an important dialogue between the 
authors of the plan and the evaluation committee. These 
interactions established evaluation as part of the plan from 
the beginning and also helped formulate and clarify the 
initial evaluation questions. During this process, the 
evaluation committee voted on specific evaluation 
questions, methods and measures. As the evaluation 
progressed, new members were added to the evaluation 
committee to provide additional guidance. 

Collection and Analysis of Data 

Perhaps the most attention has been paid to the process for 
the collection of information and the analysis of data. The 
original evaluation design included the measurement of 
outcomes for each service type resulting in several 
assessments for each child. In order to make the evaluation 
an integral part of service activity, the design needed to be 
simplified. Through meetings with groups of service 
providers and surveys of program directors, it was 
determined that too much effort was needed to collect 
multiple assessments, and it was unlikely that clients could 
distinguish between the different service types. This 
impression was confirmed through flowcharting of the 
evaluation process conducted by the evaluation committee. 
When pictures of the processes and transactions involved in 
the evaluation were analyzed, a dramatic picture of a 
complicated and cumbersome flow of activity was revealed. 
Consequently, a less complex evaluation design was created 
based on an episode of care. 

In order to reduce the burden of data collection on the 
service providers and provide managers with a minimum 
set of key outcome indicators, the measures were revised. 
Program directors were surveyed as to which measures 
were most meaningful to them. They suggested use of the 
CBCL and satisfaction forms and elimination of the 
provider completed pre-and post- treatment assessment 
forms. Concurrently, the initial measures were reviewed for 
the frequency of use in data analyses and their psychometric 
properties. It was discovered that service providers' ratings 
of treatment plan completion were infrequently used, and 
subscales of provider completed pre- and post- assessment 
measures had mixed psychometric properties. Satisfaction 
forms, however, showed good reliability and validity. 

A major effort also was made to integrate the evaluation 
into preexisting processes to minimize paperwork for 
serx'ice providers while supporting efforts to document 
compliance with standards. The evaluation was dove-tailed 
with the Department's efforts to meet defined mental health 
community standards such as continuity of care, service 
type descriptions, outcome standards, and client assessment 
and treatment plan requirements. Additionally, the 
evaluation utilized the state-wide client registration and 
assignment data base, thus automating many of the data 
collection procedures. 
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Technical assistance to the field was viewed as an essential 
part of implementation of the evaluation. As a first step, the 
evaluation committee felt that program directors and service 
providers would benefit from information on evaluation, 
data management, and how the state and local computer 
systems worked. Regional trainings were implemented to 
introduce the evaluation, followed by on-site evaluation and 
telephone training. Initially, training was provided to serve 
providers and program directors. Later, staff from medical 
records and information services were included. High 
ratings in five of six training sessions suggested that 
participants found the sessions to be informative and 
helpful in implementing work tasks. 

As the evaluation proceeded, quality control measures were 
implemented so the data used in analyses would be 
credible. The computerized data collection system was 
edited to force completion of data elements. Additionally, 
manual editing of data forms was performed and feedback 
has been given back to the centers. To further assess the 
accuracy of data, a pilot has been implemented to dovetail 
with SQA audits. 

Special emphasis was placed on empowering staff to make 
their own decisions about how to implement the evaluation 
at their own center. Efforts were made to communicate to 
the staff that the evaluation instructions were often 
guidelines and they needed to operationalize and consider 
local needs. For example, on site decision issues included 
data management and the coordination of data collection 
tasks with naturally occurring clinical activities. As a result, 
staff included pre- and post- assessments in intake and 
discharge activities, and integrated registration and history 
information into center-specific computer system data 
entry. 

Collaborating with Stakeholders 

Evaluators must collaborate with the stakeholders in using 
the results of evaluation. Managers and service providers 
need education to read and interpret data from clinical and 
administrative viewpoints. A service report was created to 
report data back to stakeholders on a regular basis, 
presenting data in a tabular fashion. Later, a text was 
produced to discuss the data, to provide suggestions on 
what the data meant, as well as how it could be used to 
monitor programs. Finally, graphic representations of trends 
were provided. 

Surveys were then implemented to determine if the 
presentation of evaluation information was adequate for the 
needs of TXMHMR managers and program directors. 
Results suggested that the service report and accompanying 
text met their needs. Additionally, informants indicated that 
preferred elements of evaluation reports included data about 
outcomes, graphics, and information in a bullet format. 
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Data about consumer satisfaction was a less frequent 
preference. In response to this feedback, a new format for 
the service reports has been developed which features more 
outcome information. In addition, the increase over time of 
requests for special data runs demonstrates that stakeholders 
are finding value in the evaluation process. 

Evaluation Results and Program Decisions 

If evaluation data are not used in decision making, then the 
evaluation has not reached its intended goal. Up to this 
point, the emphasis of the TCMHP has been placed on 
putting an evaluation system in place. Future activity must 
measure the extent to which the evaluation data are being 
used to assist in making program decisions. 

Currently, the only measure of the use of evaluation data is 
anecdotal information such as program directors' reports 
that the data have been helpful in particular instances. 
However, there has been increased contact from program 
directors and TXMHMR managers to request information, 
and TXMHMR managers have requested that special 
reports be prepared for legislative aids to support the 
funding process. 

Discussion 

The implementation of an ongoing evaluation system for 
the TCMHP has been a developmental process of 
implementation and revision, obtaining feedback from 
external and internal customers and revision again. 
Establishing interactions with the stakeholders to solve the 
problems of implementing a continuous evaluation seems 
best accomplished by starting small and expanding once 
major issues are identified. 

Clearly, the development of an evaluation process has much 
to offer to the system to be evaluated. In the present 
situation, the TCMHP evaluation has helped the program 
directors define the services they are offering and interpret 
specific aspects of the community standards. It has helped 
establish outcomes to be included in contract negotiation 
and monitoring and changed the statewide client data 
system to be more relevant to children and families served 
in the system. 

Taken together, these experiences and lessons learned seem 
to indicate that once the initial turn of the feedback loop is 
accomplished, then further reiterations of the loop are more 
easily attained. 

SumOne for Kids: Measuring and Improving Results in 
Services for Children and Families 

Pamela Meadowcroft, Ph.D. 

Introduction 
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SumOne for Kids is a multi-agency outcome monitoring 
system developed through a collaboration between The 
Pressley Ridge Center for Research and Public Policy and 
3 1 private, nonprofit child serving agencies in 
Pennsylvania. The original goal was to create a system with 
all technology supports, including functional software, all 
measurement tools and other data collection devices, 
training and audit services that was (a) low cost, (b) could 
be used at the agency or program level for program 
improvement, (c) would form the basis of a central database 
that would produce reports useful to policy makers and 
providers, and (d) would answer the questions: Who are the 
children and families we serve? What services do they 
receive at what cost? What is the impact of these services 
on their lives? To create the central database, all 
participating agencies would agree to upload a key set of 
data elements from which aggregate reports would be 
generated for comparison purposes. In this way, agencies 
could compare their own results with the combination of all 
agencies and aggregate reports could better inform 
policymakers of the results of children's services. 

The initial pilot provider agencies served over 5000 
children and families every day from all of the major 
child-serving systems including mental health, child 
welfare, juvenile justice, and special education. The types 
of services they provided also represent the full array of 
services to children and families, including in-home and 
family preservation services, adoption, day treatment and 
partial hospital programs, foster family care, therapeutic 
foster care, group homes, and residential treatment. 
Therefore, the outcome evaluation system was designed to 
be useful for all children's services and do-able by provider 
agency staff. 

The following are some of the lessons learned and values 
developed from the mistakes we made and the barriers we 
experienced in creating SumOne for Kids. 

Lesson 1 

The first rule of comedy, politics, and sex, and now 
outcome measurement, is that timing is everything. The 
original start-up for SumOne for Kids six years ago was 
painfully slow. Funding was not immediately available 
since the Foundation community at that time did not view 
outcome evaluation to be an urgent priority. Nor did the 
payers of children's services view measurement of 
outeomes as a neeessity. For example, for eight years the 
results of the direet serviees programs at Pressley Ridge had 
been evaluated by contaeting the few hundred ehildren and 
families whose serviees had been eompleted the year 
before. This follow-up evaluation served as the prototype 
for SumOne for Kids and provided Pressley Ridge 
management, elinieians, and the board of trustees with a 
way of foeusing priorities for eaeh year's program 
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development activities. However, not once in the eight 
years did the agency's referral of funding sources ask for or 
use the outcome results. Such disinterest has dramatically 
changed in the last two years. The pace and subsequent 
funding and interest in outcome evaluation in general, and 
SumOne for Kids in particular, has exploded. 

Lesson 2 

Build on what the users of the outcome monitoring system 
are already doing. SumOne for Kids staff take participating 
agencies through a design process that builds on what they 
are already collecting. The original pilot group of agencies 
helped determine all of the data elements that fully describe 
the children and families served and the types of services 
received. In replications, this same customizing process is 
used to ensure that participating groups have input into the 
data that is required by the outcome system. The pilot 
agencies also fully tested the outcome interviews that were 
developed, the forms that were used to catch the data 
elements, and the initial software. 

Lesson 3 

Make the data useful and easy to get. Most provider 
agencies already collect lots of data. But the data are only 
useful when (a) it is in a readily retrievable, readable form 
(i.e. attractive, easy to read reports); and, (b) when it can be 
compared to benchmarks so the results have meaning. 
SumOne for Kids built into the software standardized 
reports that the pilot agencies tell us are essential for the 
day to day operating of their services. The standard reports 
were designed and tested to be readable by nonresearch 
staff; hence, agencies can use the reports without research 
staff. 

Lesson 4 

There must be a strong incentive for agencies/systems to 
become accountable. The impetus for SumOne for Kids 
came in 1989 when it began to look as if a system of 
accountability might be imposed on Pennsylvania providers 
by the legislature. The provider agencies decided to be 
pro-active, develop their own system, and then turn it over 
to the state for use as a state-imposed monitoring system 
(bottom-up/top down approach). While SumOne for Kids is 
based on self-evaluation assumptions in which agencies 
want to know how effective their services are, outcome 
monitoring will require mandates in some form to insure the 
level of commitment required of agencies to produce 
accurate, timely data. Only recently has Pennsylvania begun 
to mandate outcome evaluation in different forms. For 
example, in Allegheny County, outcome evaluation is now 
part of the contracting process with all Children and Youth 
Services providers. 

Lesson 5 
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Get political support early on and throughout the 
development and implementation of any outcome 
monitoring system. An outcome monitoring system may be 
technically valid and sophisticated, but not at all useful if it 
does not have widespread political support. When SumOne 
for Kids began, a group of bureaucrats, policy makers, 
educators, researchers, and practitioners was assembled for 
the purpose of articulating a vision of what such a system 
might look like and how it might operate. Everyone present 
gave high praise to the concept, pledged the full support of 
their offices, and asked to be kept informed of project 
status. The same process has been used even more 
successfully in Maryland, where there are no regulatory 
requirements for program evaluation in residential services. 

Lesson 6 

"Outcomes" lack clear definition. Although outcomes are 
now talked about by persons at a variety of levels and in 
many kinds of systems, there is little agreement on what 
they are. Practitioners are interested in clinical outcomes 
that relate to the child's treatment plan. Program managers 
and states are more often interested in process measures, 
such as numbers of service units provided, or number of 
children served, and will consider these the "outcomes" of 
importance. Project staff, however, took a stand early-on 
that we would look at socially significant impacts of the 
services provided on the lives of children and families 
served. Such functional outcomes can appeal to the 
practitioner, program manager, as well as state level 
policy-makers. 

To assess functional outcomes requires a commitment to 
looking at the results of services after receipt of these 
services. Project staff found numerous barriers to this view 
of outcome measurement. Many providers believe their 
accountability ends when the child leaves their program. 
Others feel that post-discharge results can be useful in 
determining program change, but there is no agreement on 
how long after discharge that responsibility lasts. The 
project decided to ask a large group of varied children's 
services stakeholders to define outcomes for us, thus 
avoiding having to debate the outcomes definition with 
evaluation experts or anxious providers. 

Lesson 7 

Involve stakeholders in key aspects of developing outcome 
measures. This lesson, as all the others, was learned over 
and over again. A large-scale social validation survey of 
over 700 Pennsylvania stakeholders of children's services 
defined the outcome indicators for us. (This survey was 
reported at the Research Conference in 1992.) The survey 
asked respondents to indicate how important various issues 
were to them and how satisfied they were with the services 
available in their community to address those issues. With a 
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90% response rate, project staff felt they had solid evidence 
of what issues should be put forward as the most important 
ones to measure, which were: 

• stability and restrictiveness of children's living 
environments; 

• use of drugs and alcohol; 

• school attendance and graduation; 

• employment and job readiness; and 

• protection from harm. 

The 3 1 pilot agencies provided SumOne for Kids staff with 
the sites to test out the various tools that were designed to 
measure the above outcome indicators. A similar 
stakeholder survey was conducted in Maryland with over 
1000 participants. The results bore a striking similarity to 
that of the Pennsylvania survey. 

Lesson 8 

Generate products quickly and keep the momentum active 
or participants will lose interest, at best. A certain 
momentum must be achieved and maintained to keep 
participants involved, and there needs to be regular 
communication between project staff and its participants. 
Newsletters, reports on results of each project step, getting 
part of the data system operating and producing results right 
away are some of the tactics project staff learned along the 
way to keep project participants interested. Dispose of 
research methods that are time-consuming and opt for ones 
that, while less rigorous, will produce reasonably valid 
results with less time. 

Lesson 9 

Willima of Ochen of the middle ages gave us this lesson: 
"keep it as simple as possible, but no simpler." SumOne for 
Kids aimed to create a "simple" product&endash;one that 
was useful and easy to use. The final product is far bigger 
than originally thought necessary. It was perhaps inevitable 
because this first-of-its-kind product used a consensus 
model involving over 3 1 agencies and other stakeholders in 
the design and development. The first "final" product 
(which includes a comprehensive database on describing 
the children and families served, services received, and 
results produced in five major functional outcome areas) 
has been met with enthusiasm by those who have used it but 
with concerns by those who see it as too much. Our 
interaction with others who are developing outcome 
measurement systems indicates that the move to complex is 
quicker than the move to elegant simplicity. Future 
developments will be a balance of adding more to the 
system (such as "protective factors" and eliminating 
complicated protocols and unused data. 

Discussion 
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The project proved sufficiently successful to be spun-off 
into a separate corporation called the Corporation for 
Standards and Outcomes (CS&O). CS&O is now 
replicating and improving upon SumOne for Kids in 
Maryland through the Maryland association of 65 
child-serving agencies in that state and has made 
participation in SumOne for Kids mandatory for agency 
members. Given the multi-agency, multiple systems 
represented by these provider agencies, and statewide 
nature of the SumOne for Kids outcome measurement 
system, staff believe that the results will ultimately have a 
powerful impact on children's services in Pennsylvania and 
Maryland. 
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